{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Spacy Transformers Demo",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WFsWgat3MGCd",
        "colab_type": "text"
      },
      "source": [
        "# spaCy Transformers Demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKvZ1y3d3Ak8",
        "colab_type": "text"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lG3ReZc9ESyVPsstjuu5ek73u6vVsi3X)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zJqDMGd8MROv",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://d33wubrfki0l68.cloudfront.net/d04566d0f6671ae94fdae6fa3f767f5a6553d335/c50f0/blog/img/spacy-pytorch-transformers.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzJ2Uy6TTt58",
        "colab_type": "text"
      },
      "source": [
        "# Set-Up"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BhrT_sQMiH0",
        "colab_type": "text"
      },
      "source": [
        "Setting up the environment in Colab to run various experiments, note the cuda version of spacy-transformers is being downloaded"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SHnQiSqRZFLg",
        "colab_type": "code",
        "outputId": "e656e375-1535-4e2a-de45-ab020328284d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install gputil\n",
        "!pip install torch==1.1.0\n",
        "!pip install spacy-transformers[cuda100]==0.5.1\n",
        "!python -m spacy download en_trf_xlnetbasecased_lg\n",
        "!python -m spacy download en_trf_bertbaseuncased_lg"
      ],
      "execution_count": 0,
      "outputs": [
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1j0oHvNgMwms",
        "colab_type": "text"
      },
      "source": [
        "You will need to **restart runtime after these installs** to reinstatiate the environment/directory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qt50zr1JOe5N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import spacy\n",
        "import GPUtil\n",
        "import torch\n",
        "import numpy\n",
        "from numpy.testing import assert_almost_equal\n",
        "from scipy.spatial import distance\n",
        "import cupy\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4dh86HU_Op-_",
        "colab_type": "text"
      },
      "source": [
        "Checks whether GPU is available, switches to cuda if it is"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0MfBlpUiKVKt",
        "colab_type": "code",
        "outputId": "63b031af-42db-4a40-e819-30d4f189706d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "is_using_gpu = spacy.prefer_gpu()\n",
        "if is_using_gpu:\n",
        "    print(\"Using GPU!\")\n",
        "    torch.set_default_tensor_type(\"torch.cuda.FloatTensor\")\n",
        "    print(\"GPU Usage\")\n",
        "    GPUtil.showUtilization()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using GPU!\n",
            "GPU Usage\n",
            "| ID | GPU | MEM |\n",
            "------------------\n",
            "|  0 |  0% |  0% |\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kUzXv4DfTyml",
        "colab_type": "text"
      },
      "source": [
        "# XL-Net & BERT Models Explained"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uoy7qKqvTsgC",
        "colab_type": "text"
      },
      "source": [
        "2018 was a breakthrough year for NLP with the release of BERT , most of them centered around language modeling.  In case you’re not familiar, language modeling is a fancy word for the task of predicting the next word in a sentence given all previous words. This seemingly simple task has a surprising amount of depth and the true potential of language modeling started to be unlocked by methods using it as a pretraining method.\n",
        "\n",
        "The forerunners in this trend were ULMFiT and ELMo, both of which used LSTM-based language models. The basic idea of these methods was to train a language model on massive amounts of unlabeled data and then use the internal representations of the language model on subsequent tasks with smaller datasets such as question answering and text classification. This was a form of transfer learning, where a larger dataset was used to bootstrap a model that could then perform better on other tasks. The reason this worked so well was that language models captured general aspects of the input text that were almost universally useful. Indeed, both ULMFiT and ELMo were a massive success, producing state-of-the-art results on numerous tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zObcNBdbUq4j",
        "colab_type": "text"
      },
      "source": [
        "## BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvLRBxqzUpN6",
        "colab_type": "text"
      },
      "source": [
        "BERT stands for “Bidirectional Encoder Representations from Transformers”. It is a neural network architecture that can model bidirectional contexts in text data using Transformer.\n",
        "\n",
        "Traditional language models are trained in a left-to-right fashion to predict the next word given a sequence of words. This has the limitation of not requiring the model to model bidirectional context. What does “bidirectional context” mean? For some words, their meaning might only become apparent when you look at both the left and right context simultaneously. The simultaneous part is important: models like ELMo train two separate models that each take the left and right context into account but do not train a model that uses both at the same time.\n",
        "\n",
        "BERT solves this problem by introducing a new task in the form of masked language modeling. The idea is simple: instead of predicting the next token in a sequence, BERT replaces random words in the input sentence with the special [MASK] token and attempts to predict what the original token was. In addition to this, BERT used the powerful Transformer architecture to incorporate information from the entire input sentence.\n",
        "\n",
        "Equipped with these two approaches, BERT achieved state-of-the-art performance across numerous tasks. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NTlnu4aVQiTi",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "model_choice = \"en_trf_bertbaseuncased_lg\" #@param [\"en_trf_bertbaseuncased_lg\", \"en_trf_xlnetbasecased_lg\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qq54PBavWdF0",
        "colab_type": "text"
      },
      "source": [
        "One important detail is that BERT uses wordpieces (e.g. playing -> play + ##ing)instead of words. This is effective in reducing the size of the vocabulary and increases the amount of data that is available for each word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CCaAD1k7KiIi",
        "colab_type": "code",
        "outputId": "883b422d-ac8b-43bd-e330-2770be93840b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "nlp = spacy.load(model_choice)\n",
        "doc = nlp(\"Here is some text to encode.\")\n",
        "assert doc.tensor.shape == (7, 768)  # Always has one row per token\n",
        "print(doc._.trf_word_pieces_)  # String values of the wordpieces\n",
        "# The raw transformer output has one row per wordpiece.\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'here', 'is', 'some', 'text', 'to', 'en', '##code', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QFfXJFLSaMcd",
        "colab_type": "text"
      },
      "source": [
        "Here we can see that for the 10 word piece parts there is an individual encoding of size 768. Spacy provides a convenient utility to align the wordpieces back to the original words.  \n",
        "As the word **encode** has been split into its component parts - if we wanted to extract it's token representation as a single word we would need to pool together the 6th and 7th vector representations. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5A6p-Nn-aV5S",
        "colab_type": "code",
        "outputId": "7fcf665f-b66b-4c36-bb81-1a2d9c337a8c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "print(doc._.trf_word_pieces)  # Wordpiece IDs (note: *not* spaCy's hash values!)\n",
        "print(doc._.trf_alignment)  # Alignment between spaCy tokens and wordpieces"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[101, 2182, 2003, 2070, 3793, 2000, 4372, 16044, 1012, 102]\n",
            "[[1], [2], [3], [4], [5], [6, 7], [8]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Ct-Sxk5XOiz",
        "colab_type": "text"
      },
      "source": [
        "We don't see any masked tokens as those are used during the training batches for the model to learn word representations. As we're using pre-trained models these masks are not part of the outputs. The special [CLS] and [SEP] tokens are still output as part of the code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHdtIYVXcAc5",
        "colab_type": "text"
      },
      "source": [
        "BERT prepends a [CLS] token (short for “classification”) to the start of each sentence (this is essentially like a start-of-sentence token) and is used as an overall representation of the sentence in downstream tasks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NFcmE1mQb-99",
        "colab_type": "code",
        "outputId": "19a735a3-7a24-4a47-f9f6-6ee2f89005fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(f\"The {doc._.trf_word_pieces_[0]} token embedding can be retrieved by getting the first embedding from the output - it's the same size as the other embeddings: {len(doc._.trf_last_hidden_state[0])}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The [CLS] token embedding can be retrieved by getting the first embedding from the output - it's the same size as the other embeddings: 768\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OmXWLjrHe3Vu",
        "colab_type": "text"
      },
      "source": [
        "The last hidden state is the encoding value of the last hidden layer in the BERT architecture and can be retrieved using the *doc._.trf_last_hidden_state* method.  \n",
        "Running the method on our document gives us the embedding for each wordpiece token"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_dXJgxCW-s2",
        "colab_type": "code",
        "outputId": "cb46a251-6f5a-4e63-bb1a-597b0616940c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(doc._.trf_last_hidden_state.shape)\n",
        "assert len(doc._.trf_last_hidden_state) == len(doc._.trf_word_pieces)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10, 768)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KRQOGjDdfXnN",
        "colab_type": "text"
      },
      "source": [
        "If we wanted to retrieve every hidden layer's output the *doc._.trf_all_hidden_states* accesses a tensor containing all layers of every token  \n",
        "** At time of writing this method doesn't yet work and is a known issue in the github **"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L7OaR5LUVhE3",
        "colab_type": "code",
        "outputId": "ba5a03df-1cf5-425f-93e5-a9432d9a19ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(doc._.trf_all_hidden_states)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kmskHTjTav8_",
        "colab_type": "text"
      },
      "source": [
        "While the [CLS] token is often used as a sentence representation in downstream tasks - it's also possible to sum the component embeddings for each word to get a sentence level vector"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pPlyGbJPZ3g6",
        "colab_type": "code",
        "outputId": "5ac9ac10-287f-453e-92bf-3827334af3af",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(f\"The sentence level representation retains the same embedding dimensions using a sum-pooled vector match : {len(doc.tensor.sum(axis=0))}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The sentence level representation retains the same embedding dimensions using a sum-pooled vector match : 768\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JivgE7RibkL_",
        "colab_type": "code",
        "outputId": "7eebe2f9-31a1-4aaa-9641-1c9cb8f8e4a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "doc.tensor.sum(axis=0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 1.92071831e+00, -2.27924675e-01,  9.30034518e-02, -1.93034962e-01,\n",
              "       -8.33929181e-01, -5.17823124e+00,  1.63885760e+00,  5.26988888e+00,\n",
              "       -5.49891591e-03,  4.23406363e-01,  4.84476984e-01, -2.48546958e+00,\n",
              "       -1.97492468e+00,  1.45040047e+00, -4.58841419e+00,  1.33792830e+00,\n",
              "       -3.63066268e+00,  2.52574968e+00,  3.23240161e-02,  1.73363376e+00,\n",
              "       -9.02754664e-01, -2.40544513e-01, -5.86369324e+00,  1.33724976e+00,\n",
              "        8.09447193e+00, -2.85938358e+00, -3.24457264e+00, -1.94612670e+00,\n",
              "       -7.56774235e+00, -2.41960573e+00,  5.64183593e-01,  8.24668646e-01,\n",
              "       -3.08341694e+00, -1.93161607e-01, -1.11684406e+00, -1.17629361e+00,\n",
              "        1.48193562e+00, -9.17339146e-01, -9.93975759e-01,  2.36142230e+00,\n",
              "       -7.09139729e+00, -2.71788001e+00,  2.19503140e+00,  7.31056631e-01,\n",
              "        4.54783440e+00, -3.48939514e+00,  4.75842571e+00, -1.59263515e+00,\n",
              "       -1.60763323e-01,  6.74959958e-01, -5.97016144e+00,  3.28443575e+00,\n",
              "       -9.77768183e-01,  3.29725933e+00,  3.51441240e+00,  5.23145294e+00,\n",
              "       -3.43903661e-01, -6.91251755e+00, -3.19126987e+00,  5.01669645e-01,\n",
              "        2.59982038e+00,  3.11265802e+00, -3.67475295e+00, -5.62059689e+00,\n",
              "        3.36200094e+00, -1.07927239e+00,  7.97904432e-02,  9.19599831e-01,\n",
              "       -1.08880625e+01, -2.93374801e+00, -5.01474476e+00, -6.28118801e+00,\n",
              "       -1.63564241e+00, -2.62475467e+00, -1.40201664e+00,  3.99704695e+00,\n",
              "       -4.39286709e+00,  1.46052027e+00, -3.97154140e+00, -3.93953419e+00,\n",
              "       -1.82605982e+00,  5.51324987e+00, -1.76549745e+00,  3.08551717e+00,\n",
              "        3.69003105e+00, -1.82436967e+00, -5.76437187e+00, -1.74665260e+00,\n",
              "       -2.72758007e+00,  3.77676773e+00, -1.90001416e+00, -6.11936390e-01,\n",
              "       -8.04168999e-01,  1.27664208e+00,  4.75507736e+00, -1.27682996e+00,\n",
              "       -2.08835459e+00,  3.65541482e+00, -1.37595069e+00,  3.29652715e+00,\n",
              "        1.77457166e+00, -4.22818375e+00,  2.71131396e+00, -1.74214149e+00,\n",
              "       -2.53007126e+00, -4.76854861e-01, -1.73535419e+00,  2.82738280e+00,\n",
              "        4.36472797e+00,  1.64266419e+00,  3.61045527e+00, -4.57057571e+00,\n",
              "        4.99968261e-01, -3.65650594e-01, -5.16320515e+00,  1.17697930e+00,\n",
              "        1.36396253e+00, -8.06679010e-01,  9.45729434e-01, -3.56211209e+00,\n",
              "       -5.69730341e-01,  3.40754330e-01,  1.31558001e+00,  9.38303375e+00,\n",
              "       -1.36814702e+00,  5.01520538e+00, -3.30154181e-01,  5.16698551e+00,\n",
              "        4.08692265e+00, -2.14737368e+00,  2.23410749e+00,  7.40507221e+00,\n",
              "        5.15285969e+00, -3.73267388e+00, -1.82323813e+00,  2.91274476e+00,\n",
              "        2.01573491e+00, -2.11040163e+00, -6.66435337e+00, -7.94025481e-01,\n",
              "        3.32578206e+00,  9.60492730e-01,  4.46967602e+00, -7.84203053e-01,\n",
              "        2.98428440e+00,  4.22360086e+00, -3.84640980e+00, -2.03774595e+00,\n",
              "        1.21445322e+00,  2.94687867e+00,  3.30819321e+00, -1.01059937e+00,\n",
              "       -1.16799784e+00,  1.75079072e+00, -3.45273972e+00,  2.16675496e+00,\n",
              "        1.43309939e+00,  2.96485376e+00,  8.53186607e-01,  2.52028847e+00,\n",
              "        6.67056084e+00,  5.67987442e-01, -2.00908470e+00,  1.22618425e+00,\n",
              "        1.43696904e+00,  4.50128436e-01, -2.59065300e-01,  3.18469191e+00,\n",
              "        4.85689104e-01, -3.01131606e-01, -5.25818062e+00,  6.31151915e-01,\n",
              "        5.85256338e+00, -9.92383718e-01, -3.33766484e+00,  3.44484663e+00,\n",
              "        8.97324800e-01,  4.47280502e+00,  4.21616888e+00, -6.77111864e-01,\n",
              "       -2.19733467e+01,  4.78255177e+00, -8.10009480e-01, -4.28565502e+00,\n",
              "        2.01696944e+00, -6.72775149e-01,  2.05348182e+00, -5.62805176e+00,\n",
              "       -3.75655603e+00, -1.49089098e-01, -3.80401516e+00, -9.81009603e-01,\n",
              "       -3.41021943e+00, -2.12788296e+00,  3.98419499e+00, -4.72480965e+00,\n",
              "       -6.03775859e-01, -6.23886943e-01, -1.64400458e+00,  1.76477516e+00,\n",
              "        8.71194243e-01,  7.23116398e-02,  1.45227802e+00,  9.07019079e-01,\n",
              "        1.18739963e+00, -1.27724600e+00,  1.68838561e+00, -6.35498285e-01,\n",
              "       -3.03215694e+00,  8.34060609e-02, -4.35079050e+00,  5.76508331e+00,\n",
              "       -7.38264799e-01, -2.30859876e+00,  2.72933936e+00, -6.22139335e-01,\n",
              "        1.57774913e+00, -5.86700141e-01, -4.68656445e+00, -2.83917487e-01,\n",
              "       -4.25481021e-01, -1.16330326e+00, -4.82808018e+00,  6.58038378e+00,\n",
              "       -5.21394730e-01,  7.15868092e+00, -1.16255760e-01,  2.75693798e+00,\n",
              "        7.19152737e+00,  1.36842883e+00, -1.07972980e-01, -2.15683460e+00,\n",
              "        5.03600121e+00,  2.53713083e+00, -3.45945549e+00, -1.62613761e+00,\n",
              "       -4.76050472e+00,  2.55304241e+00,  6.86458111e-01,  1.66915715e+00,\n",
              "        7.67743111e-01,  4.39762712e-01,  4.68080616e+00, -2.18304420e+00,\n",
              "       -2.40562010e+00, -7.36788809e-01, -1.42585242e+00, -1.53161079e-01,\n",
              "        4.67393970e+00, -4.05899048e+00,  3.22338867e+00, -5.95091438e+00,\n",
              "        1.09281433e+00, -6.79128838e+00, -1.00297570e+00,  9.39649701e-01,\n",
              "       -4.01409435e+00, -2.28144789e+00, -3.70267630e-01,  4.11445522e+00,\n",
              "        4.45992321e-01,  3.20690203e+00,  7.94265866e-01,  1.64253283e+00,\n",
              "       -9.78196621e+00, -5.83843994e+00,  2.10038114e+00,  1.30369675e+00,\n",
              "        2.97164822e+00,  1.36069012e+00, -1.37880528e+00, -3.32746005e+00,\n",
              "       -3.62387753e+00, -1.44693851e-01, -3.03085637e+00,  2.10597932e-01,\n",
              "        1.01521838e+00,  6.50712967e-01,  4.14906359e+00, -1.79582715e+00,\n",
              "       -6.19434714e-02,  3.83967543e+00, -3.16072798e+00,  5.18371677e+00,\n",
              "        2.39754915e-02, -3.46709967e+00, -5.23295760e-01, -1.92537940e+00,\n",
              "       -5.65473938e+00, -5.74062729e+00, -3.87220812e+00,  1.07801926e+00,\n",
              "       -3.01134920e+00, -1.54144239e+00,  4.95082092e+00,  3.45451999e+00,\n",
              "        4.81808853e+00,  9.03786659e-01, -1.46243262e+00, -5.36369228e+00,\n",
              "       -1.27302885e-01, -2.20462942e+00,  1.68899274e+00,  3.99255991e+00,\n",
              "       -8.79824638e+00,  2.80086040e+00, -3.55970478e+00, -4.74200964e+00,\n",
              "       -2.54924355e+01, -5.03116190e-01, -7.13526368e-01, -2.24845266e+00,\n",
              "        2.05855429e-01, -4.31477833e+00, -3.87100267e+00, -4.06010103e+00,\n",
              "       -3.07949710e+00, -4.44804251e-01, -7.24208117e-01, -4.32929802e+00,\n",
              "        3.58381176e+00,  4.28281069e+00,  3.99322772e+00,  4.50070858e+00,\n",
              "        1.79578733e+00,  7.59989262e-01, -3.98393822e+00,  4.50029469e+00,\n",
              "       -1.50201952e+00, -8.04409027e-01,  2.45064378e+00, -2.67230535e+00,\n",
              "        4.67668056e+00,  5.43653488e+00, -2.92271757e+00,  2.40065217e+00,\n",
              "       -4.34136295e+00, -3.18311858e+00, -2.40658104e-01, -1.35647821e+00,\n",
              "        2.24455428e+00, -1.87819791e+00,  1.98349011e+00,  7.63348639e-01,\n",
              "       -3.00229430e+00, -3.39595604e+00,  4.47662449e+00, -4.57597828e+00,\n",
              "       -1.79709089e+00,  1.65941453e+00,  1.62861204e+00, -7.68231487e+00,\n",
              "        5.51395416e+00, -4.64114761e+00, -1.20475125e+00, -2.04864836e+00,\n",
              "        1.26880690e-01,  7.43616962e+00, -2.26973462e+00,  4.67070222e-01,\n",
              "       -2.89214015e-01,  2.27774560e-01, -1.37904239e+00,  9.38034058e-03,\n",
              "        5.29815483e+00,  3.71559525e+00, -1.63568544e+00, -1.60885572e+00,\n",
              "        1.71080709e+00, -3.76137280e+00, -2.23598719e-01, -1.20720339e+00,\n",
              "       -1.99929309e+00, -2.61421657e+00, -2.35635710e+00, -1.54649389e+00,\n",
              "       -8.62488031e-01,  4.06910062e-01, -2.80636883e+00,  2.53939772e+00,\n",
              "       -6.64681959e+00, -8.14136314e+00, -1.70077455e+00,  1.32274055e+00,\n",
              "        6.01019263e-01, -5.97445488e-01,  3.53334761e+00, -3.38260174e+00,\n",
              "       -4.38304520e+00, -8.88005066e+00, -1.48362577e+00, -1.76523614e+00,\n",
              "       -1.82323027e+00, -2.56500149e+00,  1.15548754e+00, -6.34185123e+00,\n",
              "       -2.13857961e+00, -5.85910130e+00,  3.05508232e+00,  1.75359055e-01,\n",
              "       -1.13945711e+00,  9.95938063e-01, -1.99958086e-01, -7.43531227e-01,\n",
              "        7.00612211e+00, -8.49180603e+00,  3.81900764e+00,  1.79107204e-01,\n",
              "        4.44415808e-01, -1.92847550e+00,  4.12380695e+00, -5.89289427e-01,\n",
              "       -2.19030857e-01,  2.52700758e+00, -7.28505898e+00,  1.23545170e-01,\n",
              "        1.72034597e+00,  2.65499306e+00,  1.63682067e+00, -5.37367630e+00,\n",
              "        3.86439395e+00, -1.29181993e+00, -6.70695019e+00,  1.80035186e+00,\n",
              "        2.78196096e+00,  2.43799114e+00,  1.23938894e+00,  3.85081500e-01,\n",
              "        2.84131336e+00,  6.03916454e+00, -3.13152766e+00, -1.73647857e+00,\n",
              "       -4.84976053e+00, -2.25564408e+00,  2.01312518e+00, -5.52266717e-01,\n",
              "       -1.38388228e+00, -1.97307158e+00,  1.14222735e-01, -3.09220648e+00,\n",
              "       -1.21180654e-01,  2.49168110e+00,  3.42997718e+00, -2.25318742e+00,\n",
              "       -1.49089074e+00,  3.15642881e+00,  3.39825916e+00, -4.39026982e-01,\n",
              "       -5.32705367e-01,  4.06213856e+00, -2.66128612e+00, -1.19984615e+00,\n",
              "       -2.74344611e+00,  5.71743774e+00,  2.96957374e-01,  4.10658240e-01,\n",
              "        3.15381336e+00,  4.60193038e-01, -4.99184465e+00, -5.58882892e-01,\n",
              "       -2.79437125e-01, -2.41766596e+00,  1.83237267e+00, -3.67022157e+00,\n",
              "        2.81677032e+00, -3.31816792e+00,  1.86203718e+00, -4.21204567e+00,\n",
              "       -5.48508823e-01,  5.56994247e+00,  2.23243904e+00,  3.76683235e+00,\n",
              "        2.42359638e+00,  3.63441586e+00,  1.45802665e+00, -2.43719244e+00,\n",
              "       -3.22829819e+00,  1.85594380e+00, -2.95187593e+00,  1.74937010e-01,\n",
              "        1.93035054e+00,  5.05455852e-01,  1.96698332e+00,  8.43557262e+00,\n",
              "       -4.09043431e-01, -1.93461657e-01,  2.24150801e+00,  6.93349314e+00,\n",
              "        2.40417099e+00, -1.84758449e+00, -7.31837690e-01, -3.80455732e-01,\n",
              "       -1.02353239e+00,  5.75755692e+00, -1.13063312e+00,  1.62608588e+00,\n",
              "        4.85552818e-01, -2.63864994e-01, -8.48012924e+00,  2.26738906e+00,\n",
              "       -7.33784103e+00, -3.07929420e+00, -5.30258894e+00, -4.44760227e+00,\n",
              "        1.67742038e+00, -3.46928525e+00,  4.15996170e+00, -1.65668249e+00,\n",
              "        7.15942192e+00,  7.16603458e-01, -4.94963026e+00,  5.82944155e-01,\n",
              "       -8.12486231e-01, -1.01865137e+00,  2.34889293e+00,  1.45069170e+00,\n",
              "        2.25850970e-01,  5.44021606e-01,  1.77786422e+00, -5.28192616e+00,\n",
              "       -2.83672285e+00, -6.28319263e+00,  2.05787206e+00, -2.19111025e-01,\n",
              "        3.37394285e+00,  1.18093061e+00,  2.40402365e+00,  6.14415169e-01,\n",
              "       -5.82209110e+00,  2.69666409e+00,  2.95259833e+00,  3.82881856e+00,\n",
              "       -4.18458366e+00, -2.99740076e+00, -9.39688742e-01, -1.01641202e+00,\n",
              "       -7.58284760e+00,  1.73133719e+00,  1.11848569e+00, -3.14265871e+00,\n",
              "        6.22690618e-01,  2.47175145e+00,  4.71995497e+00, -2.02616358e+00,\n",
              "       -6.19953728e+00, -3.30824924e+00,  1.26933289e+00, -9.01718140e-01,\n",
              "       -1.13279831e+00,  3.61747116e-01,  3.43950796e+00, -3.52117014e+00,\n",
              "       -1.29642880e+00, -5.36547232e+00, -3.16120362e+00,  7.11368942e+00,\n",
              "       -4.27050447e+00,  2.67967010e+00,  3.84523094e-01,  5.31123817e-01,\n",
              "       -4.48134851e+00, -3.96366549e+00,  5.01558590e+00, -2.91963410e+00,\n",
              "       -5.72447395e+00,  2.85087109e+00, -2.61309910e+00,  6.24960279e+00,\n",
              "        7.97206354e+00, -1.35962510e+00,  1.48223615e+00,  5.85888672e+00,\n",
              "        4.51712513e+00,  1.34588754e+00,  4.85850334e-01,  1.95112300e+00,\n",
              "        8.54059339e-01, -3.33984375e+00, -2.32723475e+00, -3.55858803e-01,\n",
              "        9.69198942e-02,  1.63102150e+00, -2.99816060e+00, -3.62775993e+00,\n",
              "        4.54198456e+00, -8.32461643e+00,  1.05445993e+00, -3.89940333e+00,\n",
              "        1.64764369e+00,  4.72958374e+00,  4.70964551e-01, -1.54485762e-01,\n",
              "        7.84476221e-01, -1.36690474e+00, -3.84740090e+00,  3.69135118e+00,\n",
              "        1.19266224e+00, -1.16060531e+00, -1.33347845e+00,  6.16855907e+00,\n",
              "       -5.47436142e+00,  1.15192568e+00,  8.40952814e-01, -1.57534826e+00,\n",
              "        3.49223423e+00,  1.88070893e-01,  2.31602120e+00,  9.35021698e-01,\n",
              "        1.45431101e+00,  7.79714918e+00,  1.60556877e+00, -6.24087036e-01,\n",
              "        9.16775465e-01,  2.74838781e+00, -1.62136495e-01, -1.28081846e+00,\n",
              "        1.20537066e+00,  6.03951645e+00, -3.60219002e+00, -5.44166851e+00,\n",
              "        6.72917366e+00,  4.84254789e+00, -5.09027719e+00, -1.46149600e+00,\n",
              "        2.99294138e+00, -3.20804977e+00, -2.14681816e+00,  1.12418866e+00,\n",
              "        1.65189409e+00, -3.44275212e+00,  4.59517670e+00,  2.29476786e+00,\n",
              "       -4.72148895e+00,  4.74288511e+00, -4.67934752e+00, -2.69212866e+00,\n",
              "       -2.13195395e+00,  3.97079968e+00, -5.74040532e-01,  6.36711979e+00,\n",
              "       -2.25887418e-01,  5.09000969e+00, -4.67057753e+00, -5.17026806e+00,\n",
              "        3.34388232e+00,  3.48767614e+00,  2.38891363e+00, -9.85633731e-01,\n",
              "        5.20205677e-01,  2.45024228e+00,  3.08388662e+00,  9.57857132e-01,\n",
              "        1.79250813e+00, -9.61076498e-01,  3.54907483e-01,  6.27456665e-01,\n",
              "        4.32628202e+00,  4.32047033e+00,  1.85882974e+00,  4.27694035e+00,\n",
              "       -1.44910932e-01, -4.39180702e-01,  1.10263860e+00,  2.50258040e+00,\n",
              "        5.84168148e+00,  2.85441446e+00,  1.51387691e+00, -9.92009997e-01,\n",
              "        2.77427411e+00,  4.77442801e-01,  5.20341277e-01, -2.52174902e+00,\n",
              "       -2.03088045e+00,  5.20544529e+00,  3.94292068e+00,  1.33965909e-01,\n",
              "        1.29668903e+00, -5.88522077e-01, -2.46946478e+00,  1.67412663e+00,\n",
              "        1.10253739e+00, -8.55939507e-01, -2.50330353e+00,  4.47071934e+00,\n",
              "        3.95361471e+00,  2.88988590e-01, -5.58141041e+00, -4.46413374e+00,\n",
              "       -8.34658384e-01,  7.09407330e-02, -2.06993198e+00, -3.52671218e+00,\n",
              "       -3.26713920e+00, -7.13188696e+00,  3.91415358e-02,  7.86758363e-02,\n",
              "        1.63497448e+00, -2.19377828e+00, -1.67810172e-01, -4.41853285e+00,\n",
              "        1.18243814e+00, -5.08789301e+00,  7.43657887e-01, -3.90402126e+00,\n",
              "        1.23035419e+00,  4.53563786e+00,  5.81289673e+00,  9.82828200e-01,\n",
              "       -5.86771071e-01, -2.36716747e+00, -1.27675724e+00,  1.78632307e+00,\n",
              "        1.97299051e+00,  2.74908376e+00,  3.22986484e-01, -1.02588391e+00,\n",
              "       -1.22639072e+00, -6.47749519e+00,  3.34489226e+00,  5.11536884e+00,\n",
              "       -9.47832298e+00, -1.06561661e+00, -1.20595098e+00,  1.64683342e+00,\n",
              "        1.71935856e-01, -2.17813063e+00, -5.85146725e-01,  7.13856983e+00,\n",
              "       -2.48794198e+00, -1.10630250e+00,  5.10249710e+00,  6.53547668e+00,\n",
              "       -2.45196772e+00, -1.05995864e-01,  2.08180666e+00,  4.34676218e+00,\n",
              "       -1.11777556e+00, -3.23760319e+00,  7.07936764e+00,  3.64164686e+00,\n",
              "        2.88394547e+00,  2.39885354e+00, -1.05911613e-01, -1.53665304e+00,\n",
              "       -2.78753829e+00,  1.69380283e+00, -2.68149757e+00, -3.65553570e+00,\n",
              "       -2.19431400e+00,  2.75395870e-01,  2.97403717e+00, -2.24487782e+00,\n",
              "       -8.43623352e+00, -2.28678775e+00,  1.36759496e+00, -2.67910051e+00,\n",
              "       -1.85354590e+00, -3.25318193e+00,  2.69297838e+00,  2.53265905e+00,\n",
              "       -7.56149650e-01, -2.21078110e+00,  3.97919416e-01,  2.40517735e+00,\n",
              "        2.01296067e+00, -1.19751072e+00, -2.07586765e+00,  2.66453648e+00],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZukNJrq1FP6",
        "colab_type": "text"
      },
      "source": [
        "## BERT's shortcomings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K7qmyOpQ1Bwo",
        "colab_type": "text"
      },
      "source": [
        "BERT was already a revolutionary method with strong performance across multiple tasks, but it wasn’t without its flaws. XLNet pointed out two major problems with BERT.\n",
        "\n",
        "1. The [MASK] token used in training does not appear during fine-tuning\n",
        "\n",
        "BERT is trained to predict tokens replaced with the special [MASK] token. The problem is that the [MASK] token – which is at the center of training BERT – never appears when fine-tuning BERT on downstream tasks.\n",
        "\n",
        "This can cause a whole host of issues such as:\n",
        "\n",
        "What does BERT do for tokens that are not replaced with [MASK]?\n",
        "In most cases, BERT can simply copy non-masked tokens to the output. So would it really learn to produce meaningful representations for non-masked tokens?\n",
        "Of course, BERT still needs to accumulate information from all words in a sequence to denoise [MASK] tokens. But what happens if there are no [MASK] tokens in the input sentence?\n",
        "There are no clear answers to the above problems, but it’s clear that the [MASK] token is a source of train-test skew that can cause problems during fine-tuning. The authors of BERT were aware of this issue and tried to circumvent these problems by replacing some tokens with random real tokens during training instead of replacing them with the [MASK] token. However, this only constituted 10% of the noise. When only 15% of the tokens are noised to begin with, this only amounts to 1.5% of all the tokens, so is a lackluster solution.\n",
        "\n",
        "2. BERT generates predictions independently\n",
        "\n",
        "Another problem stems from the fact that BERT predicts masked tokens in parallel. Let’s illustrate with an example: Suppose we have the following sentence.\n",
        "\n",
        "*I went to [MASK] [MASK] and saw the [MASK] [MASK] [MASK].*\n",
        "\n",
        "One possible way to fill this out is\n",
        "\n",
        "*I went to New York and saw the Empire State building.*\n",
        "\n",
        "Another way is\n",
        "\n",
        "*I went to San Francisco and saw the Golden Gate bridge.*\n",
        "\n",
        "However, the sentence\n",
        "\n",
        "*I went to San Francisco and saw the Empire State building*\n",
        "\n",
        "is not valid. Despite this, BERT **predicts all masked positions in parallel, meaning that during training**, it does not learn to handle dependencies between predicting simultaneously masked tokens. In other words, it _does not learn dependencies between its own predictions_. Since BERT is not actually used to unmask tokens, this is not directly a problem. The reason this can be a problem is that this reduces the number of dependencies BERT learns at once, making the learning signal weaker than it could be.\n",
        "\n",
        "Note that neither of these problems is present in traditional language models. Language models have no [MASK] token and generate all words in a specified order so it learns dependencies between all the words in a sentence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOLDrTLkm3jt",
        "colab_type": "text"
      },
      "source": [
        "## XL-Net"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfC36c5Jm6-t",
        "colab_type": "text"
      },
      "source": [
        "The conceptual difference between BERT and XLNet. Transparent words are masked out so the model cannot rely on them. XLNet learns to predict the words in an arbitrary order but in an autoregressive, sequential manner (not necessarily left-to-right). BERT predicts all masked words simultaneously.\n",
        "\n",
        "\n",
        "XLNet does this by introducing a variant of language modeling called “permutation language modeling”. Permutation language models are trained to predict one token given preceding context like traditional language model, but instead of predicting the tokens in sequential order, it predicts tokens in some random order. To illustrate, let’s take the following sentence as an example:\n",
        "\n",
        "I like cats more than dogs.\n",
        "\n",
        "A traditional language model would predict the tokens in the order\n",
        "\n",
        "“I”, “like”, “cats”, “more”, “than”, “dogs”\n",
        "\n",
        "where each token uses all previous tokens as context."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fpEtawVV0WLb",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://i2.wp.com/mlexplained.com/wp-content/uploads/2019/06/ezgif.com-gif-maker-1.gif?resize=447%2C170)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8d8J5a100kzC",
        "colab_type": "text"
      },
      "source": [
        "In expectation, the model should learn to model the dependencies between all combinations of inputs in contrast to traditional language models that only learn dependencies in one direction.\n",
        "\n",
        "The difference between permutation language modeling and BERT is best illustrated below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BxkIUL70oP0",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://i1.wp.com/mlexplained.com/wp-content/uploads/2019/06/Screen-Shot-2019-06-22-at-5.38.12-PM.png?resize=1024%2C567&ssl=1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "cellView": "form",
        "id": "d4r5-Lxd16Im",
        "colab": {}
      },
      "source": [
        "model_choice = \"en_trf_bertbaseuncased_lg\" #@param [\"en_trf_bertbaseuncased_lg\", \"en_trf_xlnetbasecased_lg\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "KWaPB__T16I4"
      },
      "source": [
        "You can see that the XL-Net model also has the [SEP] and [CLS] tokens like the BERT model - these are in inverse positions however."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bWDTiDUo16I5",
        "colab": {}
      },
      "source": [
        "nlp = spacy.load(model_choice)\n",
        "doc = nlp(\"Here is some text to encode.\")\n",
        "assert doc.tensor.shape == (7, 768)  # Always has one row per token\n",
        "print(doc._.trf_word_pieces_)  # String values of the wordpieces\n",
        "# The raw transformer output has one row per wordpiece.\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VlDyXzJe16JB"
      },
      "source": [
        "XL-Net doesn't use the the wordpiece model to perform tokenisation but instead uses sentencepiece which doesn't split up words into their component pieces - see encode is a single token/piece"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yIt50u1a16JC",
        "colab": {}
      },
      "source": [
        "print(doc._.trf_word_pieces)  # Wordpiece IDs (note: *not* spaCy's hash values!)\n",
        "print(doc._.trf_alignment)  # Alignment between spaCy tokens and wordpieces"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoOFAmuL7qPx",
        "colab_type": "text"
      },
      "source": [
        "Spacy provides the same functionality that we previously saw with BERT: we can access the last hidden layer of each token by using the **._.trf_last_hidden_state** method. It contains 9 embeddings of size 768 - One for each wordpiece (including the [SEP] and [CEP] special tokens)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j8i6UCwg7ph0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc._.trf_last_hidden_state.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z2dWl1eP8lAL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc._.trf_last_hidden_state"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CXiw3T6l8q24",
        "colab_type": "text"
      },
      "source": [
        "We can use a sum-pooled average to get the sentence embedding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxLzBmnq8nyA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc.tensor.sum(axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pdYETUiBcddG",
        "colab_type": "text"
      },
      "source": [
        "## SOTA powered Spacy Similarity "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "cellView": "form",
        "id": "_zMjYNXSgabE",
        "colab": {}
      },
      "source": [
        "model_choice = \"en_trf_bertbaseuncased_lg\" #@param [\"en_trf_bertbaseuncased_lg\", \"en_trf_xlnetbasecased_lg\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ag2KReTdO2D",
        "colab_type": "text"
      },
      "source": [
        "As the transformer is integrated into the normal spaCy pipeline and methods - we can use the **.similarity** method to compare vectors at both token level and at sentence level - see https://spacy.io/api/token#similarity. We can also access vectors directly using the **.vector** method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ml1Mw8g0Segr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "apple1 = nlp(\"Apple shares rose on the news.\")\n",
        "apple2 = nlp(\"Apple sold fewer iPhones this quarter.\")\n",
        "apple3 = nlp(\"Apple pie is delicious.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ANGSPcpilt1",
        "colab_type": "text"
      },
      "source": [
        "At a token level - we can see that the word Apple has different embedding representations in each context and so the similarity of Apple & Apple in each context is different. The model correctly identifies the difference between the embedding representation of the company and the fruit"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K1NUD8QGihC_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(apple1[0].similarity(apple2[0]))  # 0.73428553\n",
        "print(apple1[0].similarity(apple3[0]))  # 0.43365782"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bc5jzS0ji2aQ",
        "colab_type": "text"
      },
      "source": [
        "Similarly, this can be applied at a sentence level with the two Company related Apple sentence are more similar that the apple pie sentence is"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8xjw1bzrcDFM",
        "colab_type": "code",
        "outputId": "a1fdbfa7-3c49-4f5e-9f34-56040b2f0cdc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "print(apple1.similarity(apple2)) #0.69861203\n",
        "print(apple1.similarity(apple3)) #0.5404963"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.69861203\n",
            "0.5404965\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "melSdrH5jVjl",
        "colab_type": "text"
      },
      "source": [
        "To understand what's going on under the hood, we can manually recreate the above similarity scores using numpy & scipy methods.\n",
        "First we perform a sum-pooled vector representation of each token to get a sentence embedding as we did above. Then we convert the cupy/chainer array to a numpy array"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k2OZWp87dgRX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "a1_embedding = cupy.asnumpy(apple1.tensor.sum(axis=0))\n",
        "a2_embedding = cupy.asnumpy(apple2.tensor.sum(axis=0))\n",
        "a3_embedding = cupy.asnumpy(apple3.tensor.sum(axis=0))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIUPTcPzkBkp",
        "colab_type": "text"
      },
      "source": [
        "Similarity is defined as **1 - cosine distance** between two arrays"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXDNgXPdetVZ",
        "colab_type": "code",
        "outputId": "b8a59d51-8d4c-43e0-e09a-cf6574451ac2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(f\"Similarity between Sentence 1 and Sentence 2 is : {1 - distance.cosine(a1_embedding, a2_embedding)}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Similarity between Sentence 1 and Sentence 2 is : 0.6986120343208313\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pDlin4s0gxJh",
        "colab_type": "code",
        "outputId": "7c9de59a-e447-4092-9593-a31e2039e259",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(f\"Similarity between Sentence 1 and Sentence 3 is : {1 - distance.cosine(a1_embedding, a3_embedding)}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Similarity between Sentence 1 and Sentence 3 is : 0.5404964685440063\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gCYP4vSdm_c9",
        "colab_type": "text"
      },
      "source": [
        "# Build a Sentiment Classifier using spaCy Transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SV1dMBeqG-Kx",
        "colab_type": "text"
      },
      "source": [
        "This is a notebook version of the example found in the spaCy Transformers Github repo: https://github.com/explosion/spacy-transformers/blob/master/examples/train_textcat.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtY-e6I1jZdE",
        "colab_type": "text"
      },
      "source": [
        "**Restart the kernel prior to running this section as the memory allocation on the GPU from the previous sections will cause the code to error**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AvRUxThvGmvb",
        "colab_type": "text"
      },
      "source": [
        "Loading in additional libraries for this example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3t_a1cU4E7ru",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import thinc\n",
        "import random\n",
        "import spacy\n",
        "import GPUtil\n",
        "import torch\n",
        "from spacy.util import minibatch\n",
        "from tqdm.auto import tqdm\n",
        "import unicodedata\n",
        "import wasabi\n",
        "import numpy\n",
        "from collections import Counter"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGaOGrM2GQhS",
        "colab_type": "text"
      },
      "source": [
        "Ensuring GPU is in use: \n",
        "To run this example, ensure GPU MEM ~ 1% at start"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fYUnrNLBFRMU",
        "colab_type": "code",
        "outputId": "1451c477-ec6d-4fe1-a3af-4e1ce91b81bd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "spacy.util.fix_random_seed(0)\n",
        "is_using_gpu = spacy.prefer_gpu()\n",
        "if is_using_gpu:\n",
        "    torch.set_default_tensor_type(\"torch.cuda.FloatTensor\")\n",
        "    print(\"GPU Usage\")\n",
        "    GPUtil.showUtilization()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GPU Usage\n",
            "| ID | GPU | MEM |\n",
            "------------------\n",
            "|  0 |  0% |  1% |\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QUeI3d7NG8eW",
        "colab_type": "text"
      },
      "source": [
        "We'll use the IMDB movie database for sentiment analysis (https://ai.stanford.edu/~amaas/data/sentiment/). We've imported thinc which has the imdb dataset available as a build in method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PVDVlegnD0st",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def _prepare_partition(text_label_tuples, *, preprocess=False):\n",
        "    texts, labels = zip(*text_label_tuples)\n",
        "    cats = [{\"POSITIVE\": bool(y), \"NEGATIVE\": not bool(y)} for y in labels]\n",
        "    return texts, cats\n",
        "\n",
        "def load_data(*, limit=0, dev_size=2000):\n",
        "    \"\"\"Load data from the IMDB dataset, splitting off a held-out set.\"\"\"\n",
        "    if limit != 0:\n",
        "        limit += dev_size\n",
        "    assert dev_size != 0\n",
        "    train_data, _ = thinc.extra.datasets.imdb(limit=limit)\n",
        "    assert len(train_data) > dev_size\n",
        "    random.shuffle(train_data)\n",
        "    dev_data = train_data[:dev_size]\n",
        "    train_data = train_data[dev_size:]\n",
        "    train_texts, train_labels = _prepare_partition(train_data, preprocess=False)\n",
        "    dev_texts, dev_labels = _prepare_partition(dev_data, preprocess=False)\n",
        "    return (train_texts, train_labels), (dev_texts, dev_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PtsDDKy2Hjpe",
        "colab_type": "text"
      },
      "source": [
        "We can call the above functions to generate our training and testing data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xvNA6LTEEEk5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(train_texts, train_cats), (eval_texts, eval_cats) = load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI4ZNqIrIGBS",
        "colab_type": "text"
      },
      "source": [
        "next we'll select the transformer model we want to use to load into spacy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "cellView": "form",
        "id": "7QrsNYcaIENM",
        "colab": {}
      },
      "source": [
        "model_choice = \"en_trf_xlnetbasecased_lg\" #@param [\"en_trf_bertbaseuncased_lg\", \"en_trf_xlnetbasecased_lg\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JfsM3mIBGDAc",
        "colab_type": "code",
        "outputId": "d0540816-8364-4185-87ee-571a238274ca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "nlp = spacy.load(model_choice)\n",
        "print(nlp.pipe_names)\n",
        "print(f\"Loaded model '{model_choice}'\")\n",
        "if model_choice == \"en_trf_xlnetbasecased_lg\":\n",
        "  textcat = nlp.create_pipe(\n",
        "          \"trf_textcat\", config={\"architecture\": \"softmax_class_vector\"}\n",
        "      )\n",
        "elif model_choice == \"en_trf_bertbaseuncased_lg\":\n",
        "  textcat = nlp.create_pipe(\n",
        "          \"trf_textcat\", config={\"architecture\": \"softmax_class_vector\"}\n",
        "      )\n",
        "else: \n",
        "  print(\"Choose a supported transformer model\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['sentencizer', 'trf_wordpiecer', 'trf_tok2vec']\n",
            "Loaded model 'en_trf_xlnetbasecased_lg'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J8fhlT3WIjEU",
        "colab_type": "code",
        "outputId": "331d8a3c-4a19-4338-a5ae-bed9caaccf2d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        " # add label to text classifier\n",
        "textcat.add_label(\"POSITIVE\")\n",
        "textcat.add_label(\"NEGATIVE\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dNFWKkgtIut6",
        "colab_type": "code",
        "outputId": "3a98dafe-3445-4734-8f06-039f22bc740a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "print(\"Labels:\", textcat.labels)\n",
        "nlp.add_pipe(textcat, last=True)\n",
        "print(f\"Using {len(train_texts)} training docs, {len(eval_texts)} evaluation\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Labels: ('POSITIVE', 'NEGATIVE')\n",
            "Using 23000 training docs, 2000 evaluation\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Rh-Xnh3JL12",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# total_words = sum(len(text.split()) for text in train_texts)\n",
        "train_data = list(zip(train_texts, [{\"cats\": cats} for cats in train_cats]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vnMtV7pbOF-i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "n_iter=4\n",
        "n_texts=1000 #Changed number of texts to 75 to relieve pressue on GPU memory\n",
        "batch_size=8 #batch-szie changed to 4 to relieve pressure on GPU memory\n",
        "learn_rate=2e-5\n",
        "max_wpb=1000\n",
        "pos_label=\"POSITIVE\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eKg0EgDYjIcQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cyclic_triangular_rate(min_lr, max_lr, period):\n",
        "    it = 1\n",
        "    while True:\n",
        "        # https://towardsdatascience.com/adaptive-and-cyclical-learning-rates-using-pytorch-2bf904d18dee\n",
        "        cycle = numpy.floor(1 + it / (2 * period))\n",
        "        x = numpy.abs(it / period - 2 * cycle + 1)\n",
        "        relative = max(0, 1 - x)\n",
        "        yield min_lr + (max_lr - min_lr) * relative\n",
        "        it += 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3LaQNbUqjKBl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def evaluate(nlp, texts, cats, pos_label):\n",
        "    tp = 0.0  # True positives\n",
        "    fp = 0.0  # False positives\n",
        "    fn = 0.0  # False negatives\n",
        "    tn = 0.0  # True negatives\n",
        "    total_words = sum(len(text.split()) for text in texts)\n",
        "    with tqdm(total=total_words, leave=False) as pbar:\n",
        "        for i, doc in enumerate(nlp.pipe(texts, batch_size=batch_size)):\n",
        "            gold = cats[i]\n",
        "            for label, score in doc.cats.items():\n",
        "                if label not in gold:\n",
        "                    continue\n",
        "                if label != pos_label:\n",
        "                    continue\n",
        "                if score >= 0.5 and gold[label] >= 0.5:\n",
        "                    tp += 1.0\n",
        "                elif score >= 0.5 and gold[label] < 0.5:\n",
        "                    fp += 1.0\n",
        "                elif score < 0.5 and gold[label] < 0.5:\n",
        "                    tn += 1\n",
        "                elif score < 0.5 and gold[label] >= 0.5:\n",
        "                    fn += 1\n",
        "            pbar.update(len(doc.text.split()))\n",
        "    precision = tp / (tp + fp + 1e-8)\n",
        "    recall = tp / (tp + fn + 1e-8)\n",
        "    if (precision + recall) == 0:\n",
        "        f_score = 0.0\n",
        "    else:\n",
        "        f_score = 2 * (precision * recall) / (precision + recall)\n",
        "    return {\"textcat_p\": precision, \"textcat_r\": recall, \"textcat_f\": f_score}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yno_aYKmJHSN",
        "colab_type": "code",
        "outputId": "455065cf-aa9a-4684-c3c0-c6a02fe70040",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Initialize the TextCategorizer, and create an optimizer.\n",
        "optimizer = nlp.resume_training()\n",
        "optimizer.alpha = 0.001\n",
        "optimizer.trf_weight_decay = 0.005\n",
        "optimizer.L2 = 0.0\n",
        "learn_rates = cyclic_triangular_rate(\n",
        "    learn_rate / 3, learn_rate * 3, 2 * len(train_data) // batch_size\n",
        "    )\n",
        "print(\"Training the model...\")\n",
        "print(\"{:^5}\\t{:^5}\\t{:^5}\\t{:^5}\".format(\"LOSS\", \"P\", \"R\", \"F\"))\n",
        "\n",
        "pbar = tqdm(total=100, leave=False)\n",
        "results = []\n",
        "epoch = 0\n",
        "step = 0\n",
        "eval_every = 100\n",
        "patience = 3\n",
        "while True:\n",
        "    # Train and evaluate\n",
        "    losses = Counter()\n",
        "    random.shuffle(train_data)\n",
        "    batches = minibatch(train_data, size=batch_size)\n",
        "    for batch in batches:\n",
        "        optimizer.trf_lr = next(learn_rates)\n",
        "        texts, annotations = zip(*batch)\n",
        "        nlp.update(texts, annotations, sgd=optimizer, drop=0.1, losses=losses)\n",
        "        pbar.update(1)\n",
        "        if step and (step % eval_every) == 0:\n",
        "            pbar.close()\n",
        "            with nlp.use_params(optimizer.averages):\n",
        "                scores = evaluate(nlp, eval_texts, eval_cats, pos_label)\n",
        "            results.append((scores[\"textcat_f\"], step, epoch))\n",
        "            print(\n",
        "                \"{0:.3f}\\t{1:.3f}\\t{2:.3f}\\t{3:.3f}\".format(\n",
        "                    losses[\"trf_textcat\"],\n",
        "                    scores[\"textcat_p\"],\n",
        "                    scores[\"textcat_r\"],\n",
        "                    scores[\"textcat_f\"],\n",
        "                )\n",
        "            )\n",
        "            pbar = tqdm(total=eval_every, leave=False)\n",
        "        step += 1\n",
        "    epoch += 1\n",
        "    print(f\"epoch {epoch}\")\n",
        "    # Stop if no improvement in HP.patience checkpoints\n",
        "    if results:\n",
        "        best_score, best_step, best_epoch = max(results)\n",
        "        print(f\"best score: {best_score}  best_step : {best_step}  best epoch : {best_epoch} \")\n",
        "        print(f\"break clause: {((step - best_step) // eval_every)}\")\n",
        "        if ((step - best_step) // eval_every) >= patience:\n",
        "            break\n",
        "\n",
        "    msg = wasabi.Printer()\n",
        "    table_widths = [2, 4, 6]\n",
        "    msg.info(f\"Best scoring checkpoints\")\n",
        "    msg.row([\"Epoch\", \"Step\", \"Score\"], widths=table_widths)\n",
        "    msg.row([\"-\" * width for width in table_widths])\n",
        "    for score, step, epoch in sorted(results, reverse=True)[:10]:\n",
        "        msg.row([epoch, step, \"%.2f\" % (score * 100)], widths=table_widths)\n",
        "\n",
        "    # Test the trained model\n",
        "    test_text = eval_texts[0]\n",
        "    doc = nlp(test_text)\n",
        "    print(test_text, doc.cats)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training the model...\n",
            "LOSS \t  P  \t  R  \t  F  \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3765dbb78d2c41f38a5af478e145cadc",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(IntProgress(value=0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3Qx2iVznLB-",
        "colab_type": "text"
      },
      "source": [
        "# More information & Sources"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOcKO7T-TWLY",
        "colab_type": "text"
      },
      "source": [
        "**Sources & More information:**  \n",
        "*XL-Net explanation*  \n",
        "https://mlexplained.com/2019/06/30/paper-dissected-xlnet-generalized-autoregressive-pretraining-for-language-understanding-explained/  \n",
        "Attention is all you need  \n",
        "https://arxiv.org/abs/1706.03762"
      ]
    }
  ]
}
